Active Ranking in Practice: General Ranking Functions with Sample Complexity Bounds

This paper examines the problem of ranking a collection of objects using pairwise comparisons (rankings of two objects). In a companion paper in the regular NIPS 2011 program [1], we showed that if each object x ∈ Rd is assigned a score f(x) = ||x − r|| for some unknown r ∈ Rd, then our recently proposed active ranking algorithm can recover the ranking of the scores using about d log n selectively chosen pairwise comparisons. Here we show that this same model contains all functions of the type g(x) = wTx for some unknown w ∈ Rd, thus the same bound applies. We take advantage of this fact and use kernel methods to represent more general ranking functions. This extension includes popular ranking methods such as RankSVM, and we derive nontrivial query complexity bounds for active versions of such algorithms. The efficacy of the theory and method are demonstrated by applying our kernelized adaptive algorithm to two real datasets. 1 Problem statement Given a set of n objects Θ := {θ1, . . . , θn}, we wish to discover how an oracle ranks these objects. The ranking, denoted by σ, can be thought of as a mapping σ : {1, . . . , n} →{ 1, . . . , n} that prescribes an order σ({θi}i=1) := θσ(1) ≺ θσ(2) ≺ · · · ≺ θσ(n−1) ≺ θσ(n) (1) where θi ≺ θj means θi precedes, or is preferred to, θj in the oracle’s ranking. The ranking can be learned by querying the oracle for pairwise comparisons of objects. The primary objective here is to bound the number of pairwise comparisons needed to correctly determine the ranking when the objects (and hence rankings) satisfy certain known structural constraints. We define a ranking function to be f : Θ→ R such that θi ≺ θj ⇐⇒ f(θi) < f(θj). (2) We say two ranking functions f and g are equivalent if both ranking functions correspond to the same ranking σ. In general, there are n! ways to permute n objects and we can always find an f that obeys (2) for any desired permutation. However, we assume that the oracle’s ranking function belongs to a certain class denoted by F , which may limit the set of possible rankings. Given a set of objects Θ and a ranking function class F , we denote this constrained set of possible rankings by ΣΘ,F . While F may be uncountably infinite, because of the equivalence of ranking functions, ΣΘ,F is a subset of Sn (symmetric group over n objects) and so its cardinality |ΣΘ,F | is at most n!. 2 Main theoretical results We proposed an active approach to learning rankings in a companion paper in the NIPS 2011 conference [1]. In that paper, we show that if F := {f(θ) = ||φ(θ)− r||, r ∈ Rd} where φ : Θ → Rd is fixed and known, then we can discover a ranking selected uniformly at random from the set ΣΘ,F